I've been researching and working on implementing shadow mapping into my engine. I've been following this tutorial pretty closely but I've run into an issue.

Here are some pictures to show exactly what is wrong:

Picture 1 - You can see the cat walks are casting shadows on the floor, but they are also casting shadows on the tops of themselves.

Picture 2 - Same as picture 1 with the depth FBO texture rendered in the corner.

Picture 3 - Rendering the game from the exact position that the light source is in.

I'm guessing I need to somehow compare the fragment depth of the shadow map with the fragment depth of the normal render and if the fragment depth of the render is <= the fragment depth of the shadow then I don't apply the shadow.

That's just my guess, I really don't know if I'm even on the right track. Advice would be greatly appreciated.

I can post the code related to the shadow mapping but its' basically identical to the linked tutorials code aside from the differences in Java and C++.

I'm guessing I need to somehow compare the fragment depth of the shadow map with the fragment depth of the normal render and if the fragment depth of the render is <= the fragment depth of the shadow then I don't apply the shadow.

If you already use the shadow sampler (e.g. sampler2DShadow) , then the sampler already do comparision. The problem you encounters is quite common (self shadowing/shadow acne), besides other shadowmapping approaches, the most common trick is to use a bias. Your tutorial already handles these effects, therefor try to play around with given bias values.

double bias[] = {
0.5, 0.0, 0.0, 0.0,
0.0, 0.5, 0.0, 0.0,
0.0, 0.0, 0.5, 0.0,
0.5, 0.5, 0.5, 1.0};

Is this what you are talking about? How should I "play around them".

Or is the problem something to do with this?

		vec4 shadowCoordinateWdivide = ShadowCoord / ShadowCoord.w ;
	
		// Used to lower moiré pattern and self-shadowing
		shadowCoordinateWdivide.z += 0.0005;
	
	
		float distanceFromLight = texture2D(ShadowMap,shadowCoordinateWdivide.st).z * 2;
	
	
 		float shadow = 1.0;
 			if (ShadowCoord.w > 0.0)
 				shadow = distanceFromLight < shadowCoordinateWdivide.z ? 0.5 : 1.0;
	
  		gl_FragColor = shadow * texture2D(diffuseTexture, gl_TexCoord[0].st);

Hello

// Used to lower moiré pattern and self-shadowing
shadowCoordinateWdivide.z += 0.0005;

The bias is the small value you add to the vertex's depth in order to (I quote you) lower moiré pattern and self-shadowing

Here it is 0.0005.

What was suggested was probably to try greater values

EDIT : answer to your ... next post : OK, sorry it cannot be so easy to fix !

Hello

// Used to lower moiré pattern and self-shadowing
shadowCoordinateWdivide.z += 0.0005;

The bias is the small value you add to the vertex's depth in order to (I quote you) lower moiré pattern and self-shadowing

Here it is 0.0005.

What was suggested was probably to try greater values

Yeah I tried playing with that value before I posted. I don't know if this issue is actually "self shadowing" though. The problem is that if a triangle casts a shadow, it also has the shadow cast on it. From what I read self shadowing is when you have back facing polygons being shadowed on the same model.

Just another note, the model that is casting the shadow is having the shadows cast on it. The map itself is one large OBJ of Hangemhigh I imported. I'm using old halo maps for testing :P

Okay so I'm on the right track. I set the game to render from the same position as the light and I put in an if statement that compares the depth of the pixel in the ShadowMap with the depth of the pixel that's being rendered. If the Shadow depth is < the fragDepth then it doesn't apply the Shadow.

Heres the code and result:

http://inferno.codebrainshideout.net/Images/shadows%20d.png

		vec4 shadowCoordinateWdivide = ShadowCoord / ShadowCoord.w ;
	
		// Used to lower moiré pattern and self-shadowing
		shadowCoordinateWdivide.z += 0.0005;
	
		float shadowDepth = texture2D(ShadowMap,shadowCoordinateWdivide.st).z / texture2D(ShadowMap,shadowCoordinateWdivide.st).w;
		float fragDepth = (gl_FragCoord.z / gl_FragCoord.w)/2048;
		
		float distanceFromLight = texture2D(ShadowMap,shadowCoordinateWdivide.st).z * 2;
	
	
 		float shadow = 1.0;
 		 if (ShadowCoord.w > 0.0)
 			if (shadowDepth+0.0005 < fragDepth) //THIS IS THE NEW CHECK TO STOP SELF SHADOWING
 					shadow = distanceFromLight < shadowCoordinateWdivide.z ? 0.5 : 1.0;
	
  		gl_FragColor = shadow * texture2D(diffuseTexture, gl_TexCoord[0].st);

You can see the self shadowing is gone and from what I can tell the shadows are still being cast on the things they should. The big problem is that I can't move the camera otherwise the gl_FragCoord is not going to be the same as the ShadowMap coord.

So I need to know how to transform the gl_FragCoord to face the light source in GLSL so I can compare the depths before applying a shadow.

I think my current algorithm is to cull all away facing triangles so that ones facing the light render (just like normal rendering). I use a tiny glPolyGonOffset. This will make aliasing around the triangles facing away from the light but thats ok because the dot-product between the light vector and the triangles normal will fill it in. Self shadowing won't happen.

I already have calls to glCullFace(GL_FRONT); and glCullFace(GL_BACK); in place.

I already have a working solution I just need a way to measure the distance from the fragment to the light source. *see my previous post*

Is there a way to transform the fragment to face the light or a way to transform the fragment to normal world space and then just do a normal distance test? Halp :(

So after a couple more hours of bashing my head on the keyboard, grasping desperately for a solution I think I finally have it.

http://inferno.codebrainshideout.net/Images/shadows%20e.png

Now I need to find a way to smooth the shadows edges (possibly by blurring the edges of the shadow map)?

I ended up forcing shadows on fragments who's normal was facing away from the light. I had issues with shadow artifacts on triangles facing away from the light.